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Abstract 

We characterize the class of symmetric two-player games in which tit-for-tat cannot 
be beaten even by very sophisticated opponents in a repeated game. It turns out to 
be the class of exact potential games. More generally, there is a class of simple imi- 
tation rules that includes tit-for-tat but also imitate-the-best and imitate-if-better. 
Every decision rule in this class is essentially unbeatable in exact potential games. 
Our results apply to many interesting games including all symmetric 2x2 games, 
and standard examples of Cournot duopoly, price competition, public goods games, 
common pool resource games, and minimum effort coordination games. 
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1 Introduction 



In a repeated two-player game, tit-for-tat refers to the strategy in which a player always 
chooses the opponent's action from the previous round (Rapoport and Chammah, 1965). 
Axelrod (1980a, 1980b) observed in his famous tournaments that tit-for-tat when started 
with "cooperation" is surprisingly successful in sustaining cooperation in the prisoners' 
dilemma. Part of the success is due to the fact that it can hardly be exploited by an 
opponent who may follow various more complex decision rules, a fact already noted in 
Axelrod (1980a). Axelrod and Hamilton (1981) suggest that tit-for-tat is successful from 
an evolutionary point of view (for a critique, see Selten and Hammerstein, 1984, or Nowak 
and Sigmund, 1993). 

In this paper we extend the analysis to larger strategy sets and ask what is the 
class of symmetric two-player games in which tit-for-tat cannot be beaten by any other 
decision rule? To be precise, suppose you play a repeated symmetric game against one 
opponent. Suppose further that you know that this opponent uses tit-for-tat. Since the 
rule is deterministic, you therefore know exactly what your opponent will do in all future 
periods and how he will react to your actions. The question we pose in this paper is 
whether you can use this knowledge to exploit the tit-for-tat player in the sense that 
you achieve a higher payoff than the tit-for-tat player. We call tit-for-tat essentially 
unbeatable if in the repeated game there exists no strategy of the tit-for-tat player's 
opponent with which the opponent can obtain, in total, over a possibly infinite number 
of periods, a payoff difference that is more than the maximal payoff difference between 
outcomes in the one-period game. It turns out that tit-for-tat is essentially unbeatable 
if and only if the game is an exact potential game. Exact potential games have been 
introduced by Monderer and Shapley (1996) to show existence of and convergence to 
pure Nash equilibria. They are studied widely in the literature on learning in games 
(see Sandholm, 2010). The class of symmetric two-player exact potential games includes 
many meaningful games such as all symmetric 2x2 games, and standard examples of 
Cournot duopoly, price competition, public goods games, common pool resource games, 
and minimum effort coordination games. 

More generally, we show that there is a class of decision rules such that any rule in this 
class is essentially unbeatable if the game is a two-player symmetric exact potential game. 
This class of decision rules features variants of imitation. It includes tit-for-tat but also 
imitate-the-best (Vega-Redondo, 1997) and imitate-if-better (Duersch, Oechssler, and 
Schipper, 2012a), where the latter prescribes to mimic the opponent's action from the 
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previous period if and only if the opponent received a higher payoff in the previous period. 

To gain some intuition for our results, consider the game of "chicken" presented in 
the following payoff matrix. 

swerve straight 
swerve / 3, 3 1, 4 \ 
straight 4, 1 0, y 

What should a forward looking opponent do if she knows that she is facing a tit-for-tat 
player To beat the tit-for-tat player she can play "straight" against "swerve" once. 
But if she wants to play this again, she also has to play once "swerve" against "straight" , 
which equalizes the score. Thus, the maximum payoff difference the opponent can obtain 
against a tit-for-tat player is 3, the maximal one-period payoff differential. 

Suppose now that the imitator uses the rule "imitate if and only if the other player 
obtained a higher payoff in the previous round" and starts out with playing "swerve". 
If the opponent decides to play "straight", she will earn more than the imitator today 
but will be copied by the imitator tomorrow. From then on, the imitator will stay with 
"straight" forever. If she decides to play "swerve" today, then she will earn the same as 
the imitator and the imitator will stay with "swerve" as long as the opponent stays with 
"swerve". Suppose the opponent is a dynamic relative payoff maximizer. In that case, 
the dynamic relative payoff maximizer can beat the imitator at most by the maximal 
one-period payoff differential of 3. Now suppose the opponent maximizes the sum of her 
absolute payoffs. The best an absolute payoff maximizer can do is to play swerve forever. 
In this case the imitator cannot be beaten at all as he receives the same payoff as his 
opponent. In either case, imitation comes very close to the top-performing heuristics 
and there is no evolutionary pressure against such a heuristic. 

These results extend our recent paper Duersch, Oechssler, and Schipper (2012a), in 
which we show that imitation is subject to a money pump (i.e., can be exploited without 
bounds) if and only if the relative payoff game is of the rock-paper-scissors variety. The 
current results are stronger because they show that imitation can only be exploited with 
a bound that is equal to the payoff difference in the one-period game. Furthermore, the 
class of imitation rules considered here is significantly broader. In particular, it includes 
unconditional imitation rules like tit-for-tat. However, this comes at the cost of having 
to restrict ourselves to the class of symmetric two-player exact potential games. But as 
mentioned above, many economically relevant games satisfy this property. 

^In this paper we shall call the tit-for-tat player, or more generally, the imitator "he" and all possible 
opponents "she". 
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The behavior of learning heuristics such as imitation has previously been studied 
mostly for the case when all players use the same heuristic. For the case of imitate-the- 
best, Vega-Redondo (1997) showed that in a symmetric Cournot oligopoly with imitators, 
the long run outcome converges to the competitive output if small mistakes are allowed. 
This result has been generalized to aggregative quasisubmodular games by Schipper 
(2003) and Alos- Ferrer and Ania (2005). Huck, Normann, and Oechssler (1999), Offer- 
man, Potters, and Sonnemans (2002), and Apesteguia et al. (2007, 2010) provide some 
experimental evidence in favor of imitative behavior. In contrast to the above cited liter- 
ature, the current paper deals with the interaction of an imitator and a possibly forward 
looking, very rational and patient player. Apart from experimental evidence in Duer- 
sch, Kolb, Oechssler, and Schipper (2010) and our own paper Duersch, Oechssler, and 
Schipper (2012a) we are not aware of any work that deals with this issue. For a Cournot 
oligopoly with imitators and myopic best reply players, Schipper (2009) showed that the 
imitators' long run average payoffs are strictly higher than the best reply players' average 
payoffs. 

The article is organized as follows. In the next section, we present the model and 
provide a formal definition for being unbeatable. Section 3 introduces exact potential 
games. Necessary and sufficient conditions for tit-for-tat to be essentially unbeatable 
are given in Section H] followed by a number of potential applications. We finish with 
Section |5l where we summarize and discuss the results. 

2 Model 

We consider a symmetric two-player game (X, vr), in which both players are endowed 
with the same compact set of pure actions X. For each player, the continuous payoff 
function is denoted by tt : X x X — y M, where tt{x, y) denotes the payoff to the player 
choosing the first argument when his opponent chooses the second argument. We will 
frequently make use of the following definition. 

Definition 1 (Relative payoff game) Given a symmetric two-player game {X, ir), the 
relative payoff' game is (X, A), where the relative payoff function A : X x X — > R is 
defined by 

A{x,y) = iT{x,y) - vr(?/,x). 

Note that, by construction, every relative payoff game is a symmetric zero-sum game 
since A{x,y) = —A{y,x). 
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Next, we specify a number of different imitation rules. 



Definition 2 (Tit-for-tat) A player plays strategy tit-for-tat if he plays in each period 
t > 1 whatever his opponent did in the preceding period t — 1, i.e., yt = Xt-i. Moreover, 
we allow yo to be arbitrary^ 

While tit-for-tat is an unconditional imitation rule, in economics imitation is often 
thought to be payoff dependent. The following decision rule was studied by Vega-Redondo 
(1997). A player follows imitate-the-best if for t>l his action is given by 



and arbitrary y^ E X. Duersch, Oechssler, and Schipper (2012a) use a version of it. A 
player follows imitate-if-better if for t > 1 his action is given by 



and arbitrary yo ^ X. 

The following class of imitation rules includes all three of the above rules, tit-for-tat, 
imitate-the-best, and imitate-if-better §| 

Definition 3 (Imitation) We call a player an imitator if fort > 1, his action is given 



and arbitrary y^ G X. 

That is, the imitator always adopts the opponent's action if in the previous round 
the opponent's payoff was strictly higher than that of the imitator. In other words, if 
the imitator decides to stick to his action, the other player must have had a weakly lower 
payoff. 

^In Axelrod's tournament on the prisoners' dilemma, tit-for-tat submitted by Anatol Rapoport also 
prescribed "cooperate" as initial action (see Axelrod, 1980a, 1980b). While the prisoners' dilemma has a 
well-defined cooperative action, not all games possess such an action. We therefore consider a definition 
of tit-for-tat without restrictions on the initial action. 

^In the context of the prisoner's dilemma, the class includes strategies like two-tits- for-tat, "always 
D" . and grimm trigger but not tit-for-two-tats or Pavlov (Nowak and Sigmund, 1993). 



( {xt-i} if A(a;t_i,yt_i) > 
Vtel {xt-i,yt-i} if A(xt_i,?/t_i) = 
[ {yt-i} if A{xt^i,yt-i) <0 




xt^i if A{xt-i,yt-i) > 
yt^i otherwise 



by 
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Our aim is to determine whether there exists a strategy with which the imitator's 
opponent can obtain substantially higher payoffs than the imitator. We allow for any 
strategy of the opponent, including very sophisticated ones. In particular, the opponent 
may be infinitely patient and forward looking, and may never make mistakes. More 
importantly, she may know exactly what her opponent, the imitator, will do at all times, 
including the imitator's starting value. She may also commit to any closed loop strategy. 

Consider now a situation in which the imitator starts out with a very unfavorable 
initial action. A clever opponent who knows this initial action can take advantage of it. 
Suppose that from then on the opponent has no strategy that makes her better off than 
the imitator. Arguably, the disadvantage in the initial period should not play a role in 
the long run. This motivates the following definition. 

Definition 4 (Essentially unbeatable) We say that imitation is essentially unbeat- 
able if for any strategy of the opponent, the imitator can be beaten in total by at most the 
maximal one-period payoff differential, i.e., if for any sequence of actions by the opponent 
{xo,Xi, ...) and any initial action yo. 



where yt is defined by a specific imitation rule. 

3 Exact potential games 

The question of unbeatability of imitation is closed linked to the class of exact potential 
games (Monderer and Shapley, 1996). 

Definition 5 (Exact potential games) The symmetric game (X, vr) is an exact po- 
tential game if there exists an exact potential function P : X x X — )■ M such that for all 
y ^ X and all x, x' G xj^ 

ir{x,y) - Tr{x',y) = P{x,y) - P{x' ,y), 
Tr{x,y) - TT{x',y) = P{y, x) - P{y, x'). 

^Given the symmetry of {X,tt), the second equation plays the role usually played by the quantifier 
"for all players" in the definition of potential games. 



T 




(1) 
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Duersch, Oechssler, and Schipper (2012b) show that a symmetric two-player game is 
an exact potential game if and only if its relative payoff game is also an exact potential 
game. Furthermore, in symmetric two-player games, exact potential games have a relative 
payoff function that is additively separable, and has increasing and decreasing differences. 
All these definitions may appear to be restrictive. However, we will show below that there 
is a fairly large number of important examples that fall into this class. 

Definition 6 (Additively separable) A relative payoff function A is additively sepa- 
rable if A(x, y) = f{x) + g{y) for some functions f,g:X — > R. 

Additive separable games have been studied in Balder (1997) and Peleg (1998). 

Definition 7 (Increasing/decreasing differences) A (relative) payoff function A has 
decreasing (resp. increasing) differences on X ^ X if there exists a total order > on X 
such that for all x", x', y", y' & X with x" > x' and y" > y' , 

A{x"y) - A{x',y") < {>)A{x",y') - A{x',y'). (2) 

A is a valuation if it has both decreasing and increasing differences. 

Games with increasing differences have been introduced by Topkis (1998). 

It is interesting that in our context all of the above properties define the same class 
of games. 

Remark 1 For symmetric two-player games the following conditions are equivalent: (i) 
(X, vr) is an exact potential game, (ii) (X, A) is an exact potential game. (Hi) A has 
increasing differences, (iv) A has decreasing differences, (v) A is additively separable. 

Duersch, Oechssler, and Schipper (2012b, Theorem 20) show that (i) and (ii) are 
equivalent. There, we also show that (iii) and (iv) are equivalent for all symmetric two- 
player zero-sum games (Proposition 13). Hence, (iii) or (iv) imply that A is a valuation. 
Branzei, Mallozzi, and Tijs (2003, Theorem 1) show that (ii) is equivalent to A being a 
valuation for zero-sum games. Finally, Topkis (1998, Theorem 2.6.4.) shows equivalence 
of (v) and A being a valuation for zero-sum games. 
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4 Results 



We are ready to state our main result. 

Theorem 1 Let {X, vr) be a symmetric two-player game, where X is compact and ir is 
continuous. Tit-for-tat is essentially unbeatable if and only if {X, vr) is an exact potential 
game. 

Proof. We prove here only The converse follows directly from Proposition [T] 

below. 

For tit-for-tat to be essentially unbeatable, for any T > there must not be a limit 
cycle {xt,yt)t=o, with X]t=o ^(^*' > in which the tit-for-tat-player plays yt = Xt-i 
for t > 1 and yo = xt- 

This implies in particular that for every 3-cycle (xo,X2), (xi,Xo), (x2,Xi), (xo,X2)... it 
must hold that 

A(xo, X2) + A(xi, xo) + A(x2, xi) < 0. (3) 

Since this must hold for every 3-cycle, it must also hold for the reverse cycle (x2, Xq), (xi, X2), 
(xo,Xi), (x2,Xo), yielding 

A(a;2, xo) + A(xi, X2) + A(xo, xi) < 0. (4) 

Since {X, A) is a symmetric zero-sum game, inequalities ([3]) and (jl]) imply 

A(xo, X2) + A(xi, Xo) + A(x2, xi) = 0. 

Hence, 

A(xo, X2) + A(x2, xi) = -A(xi, Xo) = A(xo, xi) 
and thus (since A(x, x) = for any x G X), 

A(xo, X2) - A(x2, X2) = A(xo, xi) - A(x2, xi). 

Since this equation holds for any 3-cycle, we have that A is a valuation. I.e., for all 

x", x', X E X, 

A(x", x) - A(x', x) = A(x", x') - A(x', x'). 
By Remark [T] it implies that {X, n) is an exact potential game. □ 

One direction of Theorem [1] actually holds for the more general class of imitation 
rules given in Definition [31 
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Proposition 1 Let {X, vr) be a symmetric two-player game, where X is compact and vr 
is continuous. If {X, vr) is an exact potential game, then any imitation rule in the class 
given in Definitionl^is essentially unbeatable. 



Proof. By Remark [H if {X,7i) is a symmetric two-player exact potential game, then 
A is additively separable. I.e., A{x,y) = f{x) + g{y) for some functions f,g:X — > M. 
Since the relative payoff game is a symmetric zero-sum game, we have that 



and hence 



A{x,x) = f{x) + g{x) = 0, 



A{x,y) = f{x)-f{y). 



Let {xo,xi, ...) be a sequence of actions generated by the opponent's strategy, and let 
{A{xt, yt)}t=o 1 be her associated sequence of relative payoffs when the imitator follows 
an imitation rule satisfying Definition [31 We claim that 

f{xt) - f{yt+i) < 0, for all t > 0. (5) 

This follows because either the imitator imitates, i.e., yt+i = Xt or he does not, yt+i = yt, 
in which case it must have been the case that A{xt,yt) = f{xt) — fijjt) < 0. 

Given ([5]), the sum of relative payoffs satisfies for any T > 0, 

T T 

VA(xt,?/t) = ^(/(xt) - f{yt)) < fixr) - f{yo) = A(xt,|/o) < maxA(x,?/), 

' ' x,y 

t=0 t=0 

where maxa. A(x, y) exists because vr is continuous and X is compact. □ 



In contrast to tit-for-tat, the existence of an exact potential function is not a necessary 
condition for being essentially unbeatable for imitate-the-best and imitate-if-better. That 
is, going from tit-for-tat to the more general class of imitation rules comes at the cost of 
losing the converse of Theorem [H To see this, consider the following game, which is not 
an exact potential game. 

ABC ABC 

0,0 0,-1 -1,0 \ A- ^ f ^ ^ \ 
-1,0 0,0 0,10 ~ 5 -1 -10 

0,-1 10,0 0,0 / C \ 1 10 / 

It is easy to see that imitate-the-best and imitate-if-better are essentially unbeatable 
for this game. However, tit-for-tat could be exploited without any bound by following 
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a cycle of actions {A ^ B ^ C A. . .). The reason for this difference is that an 
imitate-the-best or imitate-if-better player would never leave action C whereas a tit-for- 
tat player can be induced to follow the opponent from C to A. 

In the chicken game discussed in the Introduction, imitation was essentially unbeat- 
able since the maximal payoff difference was 3. Axelrod (1980a, b) observed that tit-for- 
tat was unexploitable by other decision rules in the prisoners' dilemma. More generally, 
since every symmetric 2 x 2 is an exact potential game. Proposition [T]implies the following 
corollary. 

Corollary 1 In any symmetric 2x2 game, imitation is essentially unbeatable. 

It is also easy to see why the corollary is true without making use of the notion of exact 
potential. Let X = {x, x'}. Consider a period t in which the opponent achieves a strictly 
positive relative payoff, A(x, x') > 0. (If no such period t in which the opponent achieves a 
strictly positive relative payoff exists, then trivially imitation is essentially unbeatable.) 
The relative payoff game is symmetric zero-sum and hence A{x',x) = —A{x,x') and 
A{x,x) = A{x',x') = 0. Therefore, the action combination {x,x') is the only one that 
makes imitation worse off. However, in this case, imitation will immediately imitate and 
play X in the next round. The only way to move imitation back to play x' is for the 
opponent to playing x' first, leading to the payoff A(x',x) = —A{x,x') < 0. Thus, every 
period with a positive relative profit for the opponent must be preceded by a period with 
a negative relative profit of the same absolute value. Depending on the exact imitation 
rule, the imitator might not follow in the first period where the opponent plays x', leading 
to even worse relative profits, but it will always follow to x right away. 

Note that "Matching pennies" is not a counter-example since it is not symmetric. 

A sufficient condition for the additive separability of relative payoffs and thus the 
existence of an exact potential for the symmetric two-player game {X, vr) is provided in 
the next result. 

Corollary 2 Consider a game {X, vr) with a compact action set X and a payoff' function 
that can be written as 7r(x, y) = f{x) + g{y) + a{x, y) for some continuous functions 
f,g:X — > M and a symmetric function a : X x X — > M (i.e., a{x,y) = a{y,x) for all 
x,y & X). Then imitation is essentially unbeatable. 

The following examples demonstrate that the assumption of additively separable rel- 
ative payoffs is not as restrictive as may be thought at first glance. 
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Example 1 (Cournot Duopoly with Linear Demand) Consider a (quasi) Cournot 
duopoly given by the symmetric payoff function 7t{x, y) = x{b — x — y) — c{x) with b > 0. 
Since 7i{x,y) can be written as 7i{x,y) = bx — x"^ — c{x) — xy, Corollary [2] applies, and 
imitation is essentially unbeatable. 

Example 2 (Bertrand Duopoly with Product Differentiation) Consider a differ- 
entiated duopoly with constant marginal costs, in which firms 1 and 2 set prices x and y, 
respectively. Firm I's profit function is given by 7r(x, y) = {x — c)(a + by — |x), for a > 0, 
b G [0, 1/2). Since 7r(x, y) can be written as 7r(x, y) = ax — ac + \cx — — bey + bxy, 
Corollary |2] applies, and imitation is essentially unbeatable. 

Example 3 (Public Goods) Consider the class of symmetric public good games de- 
fined by TT{x,y) = g{x,y) — c(x) where g{x,y) is some symmetric monotone increasing 
benefit function and c(x) is an increasing cost function. Usually, it is assumed that g is an 
increasing function of the sum of provisions, x + y. Various assumptions on g have been 
studied in the literature such as increasing or decreasing returns. In any case. Corollary |2] 
applies, and imitation is essentially unbeatable. 

Example 4 (Common Pool Resources) Consider a common pool resource game 
with two appropriators. Each appropriator has an endowment e > that can be in- 
vested in an outside activity with marginal payoff c > or into the common pool re- 
source. Let X G X C [0, e] denote the opponent's investment into the common pool 
resource (likewise y denotes the imitator's investment). The return from investment into 
the common pool resource is ^^(a(x + y) — b{x + y)^), with a,b > 0. So the sym- 
metric payoff function is given by 7i{x,y) = c(e — x) + ^^(a(x + y) — b{x + yY) if 
x,y > and ce otherwise (see Walker, Gardner, and Ostrom, 1990). Since A{x,y) = 
(c(e — x) + ax — bx^) — (c(e — y) + ay — by'^), Proposition [T] implies that imitation is 
essentially unbeatable. 

Example 5 (Minimum Effort Coordination) Consider the class of minimum effort 
games given by the symmetric payoff function 7r(x, y) = min{a;, y} — c{x) for some cost 
function c(-) (see Bryant, 1983, and Van Huyck, Battalio, and Beil, 1990). Corollary [2] 
implies that imitation is essentially unbeatable. 

Example 6 (Synergistic Relationship) Consider a synergistic relationship among two 
individuals. If both devote more effort to the relationship, then they are both better off. 
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but for any given effort of the opponent, the return of the player's effort first increases 
and then decreases. The symmetric payoff function is given by n{x, y) = x{c + y — x) 
with c > and x,y E X C M+ with X compact (see Osborne, 2004, p. 39). Corollary |2] 
implies that imitation is essentially unbeatable. 

Example 7 (Diamond's Search) Consider two players who exert effort searching for 
a trading partner. Any trader's probability of finding another particular trader is pro- 
portional to his own effort and the effort by the other. The payoff function is given 
by 7r(x, ?/) = axy — c{x) for a > and c increasing (see Milgrom and Roberts, 1990, 
p. 1270). The relative payoff game of this two-player game is additively separable. By 
Proposition [1] imitation is essentially unbeatable. 

5 Discussion 

We have shown in this paper that there is a class of imitation rules that is surprisingly 
robust to exploitation by any strategy in symmetric two-player exact potential games. 
This includes strategies by truly sophisticated opponents. The property that imitation is 
unbeatable in these games seems to be unique among commonly used learning rules. We 
are not aware of other rules outside our class of imitation rules share this property. For 
example, many commonly used belief learning rules, such as best response learning or 
fictitious play, can easily be exploited in all games in which a Stackelberg leader achieves 
a higher payoff than the follower (as e.g. in Cournot games). Against such rules, the 
opponent can simply stubbornly choose the Stackelberg leader action knowing that the 
belief learning player will eventually converge to the Stackelberg follower action. Thus, 
belief learning rules can be beaten without bounds in such games. Yet, it remains an 
open question for future research whether there are other behavioral rules that perform 
equally well as imitation. 

In Duersch, Oechssler, and Schipper (2012a) we show some extensions to more general 
classes of two-player games including relative payoff games with a generalized ordinal 
potential that come at cost of weakening the criterion of essentially unbeatable and 
focusing on the rule "imitate-if-better" . Extensions to ra-player games must be left for 
further research. 
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